Statistical and constraint-based taggers for French

In this paper we compare two competing approaches to part-of-speech tagging, statistical and constraint-based disambigua-tion, using French as our test language. We imposed a time limit on our experiment: the amount of time spent on the design of our constraint system was about the same as the time we used to train and test the easy-to-implement statistical model. We describe the two systems and compare the results. The accuracy of the statistical method is reasonably good, comparable to taggers for English. But the constraint-based tagger seems to be superior even with the limited time we allowed ourselves for rule development. 1 Overview In this paper we compare two competing approaches to part-of-speech tagging, statistical and constraint-based disambiguation, using French as our test language. The process of tagging consists of three stages: tokenisation, morphological analysis and dis-ambiguation. The two taggers include the same to-keniser and morphological analyser. The tokeniser uses a nite-state transducer that reads the input and outputs a token whenever it has read far enough to be sure that a token is detected. The morphological analyser contains a transducer lexicon. It produces all the legitimate tags for words that appear in the lexicon. If a word is not in the lexicon, a guesser is consulted. The guesser employs another nite-state transducer. It reads a token and prints out a set of tags depending on preexes, in-ectional information and productive endings that it nds. We make even more use of transducers in the constraint-based tagger. The tagger reads one sentence at a time, a string of words and alternative tags, feeds them to the grammaticaltransducers that remove all but one alternative tag from all the words on the basis of contextual information. If all the transducers described above (tokeniser, morphological analyser and disambiguator) could be composed together, we would get one single transducer that transforms a raw input text to a fully disambiguated output. The statistical method contains the same to-keniser and morphological analyser. The disam-biguation method is a conventional one: a hidden Markov model. 2 Morphological analysis and guessing 2.1 Morphological analyser The morphological analyser is based on a lexical transducer (Karttunen et al., 1992; Karttunen, 1994). The source lexicon and rules are represented as in the two-level model (Koskenniemi, 1983). They are compiled into a single nite-state transducer using Xerox lexical tools (Karttunen and Beesley, 1992; Karttunen, 1993). The transducer maps each innected surface form of a word …